L23

Optimization in ML

Perceptrons Review

Hill Climbing

Gradients

n Dimensional Optimization (Gradient Ascent)

Batch Gradient Ascent on the Log Likelihood Objective

\[ w\leftarrow{w+{\alpha}*\sum_{i}{\nabla}{\log}(\mathbb{P}(y^{(i)}|x^{(i)};w))} \] \[ \max_{w}ll(w)=\max_{w}g(w) \]

Stochastic Gradient Ascent on the Log Likelihood Objective

Mini-Batch Gradient Ascent on the Log Likelihood Objective

Batch Sizes

Screenshot_2023-11-14_at_7.50.07_PM.png - Multilayer network with batches: Screenshot_2023-11-14_at_7.50.56_PM.png